Benchmark Data and Evaluation Framework for Intent Discovery Around COVID-19 Vaccine Hesitancy

The COVID-19 pandemic has made a huge global impact and cost millions of lives. As COVID-19 vaccines were rolled out, they were quickly met with widespread hesitancy. To address the concerns of hesitant people, we launched VIRA, a public dialogue system aimed at addressing questions and concerns surrounding the COVID-19 vaccines. Here, we release VIRADialogs, a dataset of over 8k dialogues conducted by actual users with VIRA, providing a unique real-world conversational dataset. In light of rapid changes in users' intents, due to updates in guidelines or in response to new information, we highlight the important task of intent discovery in this use-case. We introduce a novel automatic evaluation framework for intent discovery, leveraging the existing intent classifier of VIRA. We use this framework to report baseline intent-discovery results over VIRADialogs, that highlight the difficulty of this task. © 2023 Association for Computational Linguistics.

Preliminary Diagnosis of COVID-19 using Speech Processing Techniques

This project aims to devise an alternative for Coronavirus detection using various audio signals. The aim is to create a machine-learning model assisted by speech processing techniques that can be trained to distinguish symptomatic and asymptomatic Coronavirus cases. Here the features exclusive to the vocal cord of a person is used for covid detection. The procedure is to train the classifier using a data set containing data of people of various ages both infected and disease-free, including patients with comorbidities. We presented a machine learning-based Coronavirus classifier model that can separate Coronavirus positive or negative patients from cough, breathing, and speech recordings. The model was trained and evaluated using several machine learning classifiers such as Random Forest Classifier, Logistic Regression (LR), Decision Tree Classifier, k-nearest Neighbour (KNN), Naive Bayes Classifier, Linear Discriminant Analysis, and a neural network. This project helps track COVID-19 patients at a low cost using a non-contactable procedure and reduces the workload on testing centers. © 2023 IEEE.

MedConQA: Medical Conversational Question Answering System based on Knowledge Graphs

The medical conversational system can relieve doctors' burden and improve healthcare effi-ciency, especially during the COVID-19 pan-demic. However, the existing medical dialogue systems have die problems of weak scalability, insufficient knowledge, and poor controlla-bility. Thus, we propose a medical conversa-tional question-answering (CQA) system based on the knowledge graph, namely MedConQA, which is designed as a pipeline framework to maintain high flexibility. Our system utilizes automated medical procedures, including medi-cal triage, consultation, image-text drug recom-mendation, and record. Each module has been open-sourced as a tool, which can be used alone or in combination, with robust scalability. Besides, to conduct knowledge-grounded dia-logues with users, we first construct a Chinese Medical Knowledge Graph (CMKG) and col-lect a large-scale Chinese Medical CQA (CM-CQA) dataset, and we design a series of meth-ods for reasoning more intellectually. Finally, we use several state-of-the-art (SOTA) tech-niques to keep the final generated response more controllable, which is further assured by hospital and professional evaluations. We have open-sourced related code, datasets, web pages, and tools, hoping to advance future research. © 2022 Association for Computational Linguistics.

Netnography of Social Media Addresses on COVID-19

Healthcare professionals' harness social media to encourage responsible behaviour during the COVID-19 pandemic. As internet users often struggle assessing the veracity of the information in these addresses, acoustic characteristics of the presenters' speech may play a significant role in their persuasiveness impact. Using a netnographic approach, we studied YouTubers' reactions to explore the persuasiveness attributes of COVID-19 related speeches included in YouTube videos within a South Africa context. The persuasiveness index was computed from the view count, likes and dislikes of 314 speech segments from YouTube interviews related to COVID-19. Standard acoustic features - Mel frequency cepstral coefficients - of the interviewees' voice were extracted through speech processing. Recurrent neural networks were optimized and evaluated the strength of these acoustic features to classify and predict the persuasiveness index. The cepstral feature set yielded a balanced accuracy of 86.8% and F1 score of 85.0%. These preliminary results exhibit the potential of the vocal cepstrum as predictor of persuasiveness in healthcare addresses on responsible behaviour during the COVID-19 pandemic. The results imply that quantitative acoustic analysis of a presenter's voice, independent from text, can explain the impact of social media addresses. © The Authors, (2022). All Rights Reserved. No reproduction, copy or transmission may be made without written permission from the individual authors.

Studying the Effect of Face Masks in Identifying Speakers using LSTM

During the COVID-19 pandemic, it has been a standard procedure for people all around the world to use Respiratory Protection Masks (RPM) that cover both the nose and the mouth. The Consequences of wearing RPMs, those pertaining to the perception and production of spoken communication, are rapidly becoming more prominent. Nevertheless, the utilization of face masks also causes attenuation in voice signals, and this alteration affects speech-processing technologies such as Automatic Speaker Verification (ASV) and speech-to-text conversion. An intervention by a deep learning-based algorithm is considered vital to remedy the issue of inappropriate exploitation of speaker-based technology. Therefore, in the proposed framework, a speaker identification system has been implemented to examine the effect of masks. First, the speech signals have been captured, pre-processed, and augmented by a variety of data augmentation techniques. Afterward, different 'Mel-Frequency Cepstral Coefficients' (MFCC) features have been extracted to be fed into a 'Long Short-Term Memory' (LSTM) for identifying speakers. The system's overall performance has been assessed using accuracy, precision, recall, and Fl-score, which yields 93%, 93.3%, 92.2%, and 92.8%, respectively. The obtained results are still in a rudimentary phase, and they are subjected to further enhancements in the future by data expansion and exploitation of multiple optimization techniques. © 2022 IEEE.

Intelligent mask recognition and voice dialogue system based on Raspberry Pi control

As New Coronavirus continues to mutate, virulence and infectivity are constantly strengthened. 2019-nCoV has become the most deadly virus in recent years. Its main transmission route is droplet transmission. Wearing masks can effectively block the spread of viruses. Mask is an important defense line to prevent respiratory infectious diseases, which can reduce the risk of New Coronavirus infection. The mask can not only prevent the patient from spraying droplets and reduce the amount and speed of droplets.but also block the droplet core containing virus and prevent the wearer from inhaling. In public places.especially in places with large passenger flow, it is a drop in the bucket to rely solely on manual supervision of customers wearing masks. In order to solve this problem.an intelligent mask detection system is designed, It supports the corresponding models of all mainstream frameworks for face mask detection (pytorch, tensorflow, keras,mxnet and cafe) (the model trained by keras and other framework models converted),and provides the reasoning code of all five frameworks. At the same time.it also has the voice dialogue function,which can wake up the voice robot in real time and achieve man-machine communication. Let inspectors feel more at ease and consumers feel more at ease. © 2022 IEEE.

BU-TTS: An Open-Source, Bilingual Welsh-English, Text-to-Speech Corpus

This paper presents the design, collection and verification of a bilingual text-to-speech synthesis corpus for Welsh and English. The ever expanding voice collection currently contains almost 10 hours of recordings from a bilingual, phonetically balanced text corpus. The speakers consist of a professional voice actor and three amateur contributors, with male and female accents from north and south Wales. This corpus provides audio-text pairs for building and training high-quality bilingual Welsh-English neural based TTS systems. We describe the process by which we created a phonetically balanced prompt set and the challenges of attempting to collate such a dataset during the COVID-19 pandemic. Our initial findings in validating the corpus via the implementation of a state-of-the-art TTS models are presented. This corpus represents the first open-source Welsh language corpus large enough to capitalise on neural TTS architectures. © European Language Resources Association (ELRA)

A novel multimodal fusion framework for early diagnosis and accurate classification of COVID-19 patients using X-ray images and speech signal processing techniques.

BACKGROUND AND OBJECTIVE: COVID-19 outbreak has become one of the most challenging problems for human being. It is a communicable disease caused by a new coronavirus strain, which infected over 375 million people already and caused almost 6 million deaths. This paper aims to develop and design a framework for early diagnosis and fast classification of COVID-19 symptoms using multimodal Deep Learning techniques. METHODS: we collected chest X-ray and cough sample data from open source datasets, Cohen and datasets and local hospitals. The features are extracted from the chest X-ray images are extracted from chest X-ray datasets. We also used cough audio datasets from Coswara project and local hospitals. The publicly available Coughvid DetectNow and Virufy datasets are used to evaluate COVID-19 detection based on speech sounds, respiratory, and cough. The collected audio data comprises slow and fast breathing, shallow and deep coughing, spoken digits, and phonation of sustained vowels. Gender, geographical location, age, preexisting medical conditions, and current health status (COVID-19 and Non-COVID-19) are recorded. RESULTS: The proposed framework uses the selection algorithm of the pre-trained network to determine the best fusion model characterized by the pre-trained chest X-ray and cough models. Third, deep chest X-ray fusion by discriminant correlation analysis is used to fuse discriminatory features from the two models. The proposed framework achieved recognition accuracy, specificity, and sensitivity of 98.91%, 96.25%, and 97.69%, respectively. With the fusion method we obtained 94.99% accuracy. CONCLUSION: This paper examines the effectiveness of well-known ML architectures on a joint collection of chest-X-rays and cough samples for early classification of COVID-19. It shows that existing methods can effectively used for diagnosis and suggesting that the fusion learning paradigm could be a crucial asset in diagnosing future unknown illnesses. The proposed framework supports health informatics basis on early diagnosis, clinical decision support, and accurate prediction.

Performance Comparison of Oral, Laryngeal and Thoracic Sounds in the Detection of COVID-19 by Employing Machine Learning Techniques

COVID-19 can directly or indirectly cause lung involvements by crossing the upper airways. It is essential to quickly detect the lung involvement condition and to follow up and treat these patients by early hospitalization. In recent COVID-19 diagnosis procedure, PCR testing is applied to the samples taken from the patients and a quarantine period is applied to the patient until the test results are received. As a complement to PCR tests and for faster diagnosis, thin-section lung computed tomography (CT) imaging is used in COVID-19 patients. In this study, it is aimed to develop a method that is as reliable as CT, and compared to CT, less risky, more accessible, and less costly for the diagnosis of COVID-19 disease. For this purpose, first speech and cough sounds from the oral, laryngeal and thoracic regions of COVID-19 patients and healthy individuals were obtained with the multi-channel voice recording system we proposed, the obtained data were processed with machine learning methods and their accuracies in COVID-19 diagnosis were presented comparatively. In our study, the best results were obtained with the features extracted from the cough sounds taken from the oral region. © 2022 IEEE.

INTELLIBOT - Intelligent Voice Assisted Chatbot with Sentiment Analysis, COVID Dashboard and Offensive Text Detection

Chatbot has become an essential crowd puller in the world today and is used in various domains and professions. With increasing technologies and advancements in AI, components of voice assistance have been gaining prolific importance when integrated with chatbots. INTELLIBOT is a smart bot that not only interacts with its users through an interactive and aesthetic platform but also had added features for customized experience. It makes use of speech to text and text to speech processing to listen to the user and speak back to them. This paper would give insights on the various applications of chatbots and existing systems along with the system we have proposed to overcome and curb the challenges posed by them through the INTELLIBOT framework. Further, the paper would elucidate the use of Naïve-Bayes algorithm and pattern matching algorithms for the same. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Automated Recognition of Hindi Word Audio Clips for Indian Children using Clustering-based Filters and Binary Classifier

Speech recognition systems have made remarkable progress in the last few decades but most of the work has been done for adult speech. The rise of online learning during Covid-19 pandemic highlights the need for voice-enabled assistants for children so that they can navigate the menus and interfaces seamlessly. Speech recognition for children will also be very useful to develop automated reading assessment tools. However, such technology for children is challenging for a country like India where differences in accents, diction and enunciation is significant but available children speech data is limited. Through this paper, I tried various approaches to recognize hindi word audios. Commercially available Google Speech-to-Text performs poorly with only 49.7% accuracy at recall of 0.24 while recognising audio samples containing hindi words spoken by children. Using the same dataset, I experimented with clustering algorithm and logistic regression and found that the accuracy improves upto 81% with logistic regression. The paper also highlights the importance of data preprocessing by performing noise reduction using Butterworth low pass filters. © ICNLSP 2021. All Rights Reserved.

A Hybrid Sparrow Search Algorithm of the Hyperparameter Optimization in Deep Learning

Deep learning has been widely used in different fields such as computer vision and speech processing. The performance of deep learning algorithms is greatly affected by their hyperparameters. For complex machine learning models such as deep neural networks, it is difficult to determine their hyperparameters. In addition, existing hyperparameter optimization algorithms easily converge to a local optimal solution. This paper proposes a method for hyperparameter optimization that combines the Sparrow Search Algorithm and Particle Swarm Optimization, called the Hybrid Sparrow Search Algorithm. This method takes advantages of avoiding the local optimal solution in the Sparrow Search Algorithm and the search efficiency of Particle Swarm Optimization to achieve global optimization. Experiments verified the proposed algorithm in simple and complex networks. The results show that the Hybrid Sparrow Search Algorithm has the strong global search capability to avoid local optimal solutions and satisfactory search efficiency in both low and high-dimensional spaces. The proposed method provides a new solution for hyperparameter optimization problems in deep learning models.

COVID-19 Diagnosis from Cough Acoustics using ConvNets and Data Augmentation

With the periodic rise and fall of COVID-19 and countries being inflicted by its waves, an efficient, economic, and effortless diagnosis procedure for the virus has been the utmost need of the hour. Amongst the infected subjects, the asymptomatic ones need not be entirely free of symptoms caused by the virus. They might not show any observable symptoms like the symptomatic subjects, but they may differ from uninfected ones in the way they cough. These differences in the coughing sounds are minute and indiscernible to the human ear, however, these can be captured using machine learning models. In this paper, we present a deep learning approach to analyze the acoustic dataset provided in Track 1 of the DiCOVA 2021 Challenge containing cough sound recordings belonging to both COVID-19 positive and negative examples. To perform the classification we propose a ConvNet model. It achieved an AUC score percentage of 72.23 on a blind test set provided in the challenge for an unbiased evaluation of the models. Moreover, the ConvNet model incorporated with Data Augmentation further increased the AUC score percentage from 72.23 to 87.07. It also outperformed the DiCOVA 2021 Challenge's baseline model by 23% thus, claiming the top position on the DiCOVA 2021 Challenge leaderboard. This paper proposes the use of Mel Frequency Cepstral Coefficients as the input features to the proposed model. © 2021 IEEE.

A privacy-preserving dialogue system based on argumentation

Dialogue systems are a class of increasingly popular AI-based solutions to support timely and interactive communication with users in many domains. Due to the apparent possibility of users disclosing their sensitive data when interacting with such systems, ensuring that the systems follow the relevant laws, regulations, and ethical principles should be of primary concern. In this context, we discuss the main open points regarding these aspects and propose an approach grounded on a computational argumentation framework. Our approach ensures that user data are managed according to data minimization, purpose limitation, and integrity. Moreover, it is endowed with the capability of providing motivations for the system responses to offer transparency and explainability. We illustrate the architecture using as a case study a COVID-19 vaccine information system, discuss its theoretical properties, and evaluate it empirically. © 2022 The Author(s)

Machine Learning Methods for Signal, Image and Speech Processing

The signal processing (SP) landscape has been enriched by recent advances in artificial intelligence (AI) and machine learning (ML), yielding new tools for signal estimation, classification, prediction, and manipulation. Layered signal representations, nonlinear function approximation and nonlinear signal prediction are now feasible at very large scale in both dimensionality and data size. These are leading to significant performance gains in a variety of long-standing problem domains like speech and Image analysis. As well as providing the ability to construct new classes of nonlinear functions (e.g., fusion, nonlinear filtering). This book will help academics, researchers, developers, graduate and undergraduate students to comprehend complex SP data across a wide range of topical application areas such as social multimedia data collected from social media networks, medical imaging data, data from Covid tests etc. This book focuses on AI utilization in the speech, image, communications and yirtual reality domains. © 2021 River Publishers. All rights reserved.

Speech Processing as a Far-Transfer Gauge of Serious Games for Cognitive Training in Aging: Randomized Controlled Trial of Web-Based Effectivate Training.

BACKGROUND: The number of serious games for cognitive training in aging (SGCTAs) is proliferating in the market and attempting to combat one of the most feared aspects of aging-cognitive decline. However, the efficacy of many SGCTAs is still questionable. Even the measures used to validate SGCTAs are up for debate, with most studies using cognitive measures that gauge improvement in trained tasks, also known as near transfer. This study takes a different approach, testing the efficacy of the SGCTA-Effectivate-in generating tangible far-transfer improvements in a nontrained task-the Eye tracking of Word Identification in Noise Under Memory Increased Load (E-WINDMIL)-which tests speech processing in adverse conditions. OBJECTIVE: This study aimed to validate the use of a real-time measure of speech processing as a gauge of the far-transfer efficacy of an SGCTA designed to train executive functions. METHODS: In a randomized controlled trial that included 40 participants, we tested 20 (50%) older adults before and after self-administering the SGCTA Effectivate training and compared their performance with that of the control group of 20 (50%) older adults. The E-WINDMIL eye-tracking task was administered to all participants by blinded experimenters in 2 sessions separated by 2 to 8 weeks. RESULTS: Specifically, we tested the change between sessions in the efficiency of segregating the spoken target word from its sound-sharing alternative, as the word unfolds in time. We found that training with the SGCTA Effectivate improved both early and late speech processing in adverse conditions, with higher discrimination scores in the training group than in the control group (early processing: F1,38=7.371; P=.01; ηp2=0.162 and late processing: F1,38=9.003; P=.005; ηp2=0.192). CONCLUSIONS: This study found the E-WINDMIL measure of speech processing to be a valid gauge for the far-transfer effects of executive function training. As the SGCTA Effectivate does not train any auditory task or language processing, our results provide preliminary support for the ability of Effectivate to create a generalized cognitive improvement. Given the crucial role of speech processing in healthy and successful aging, we encourage researchers and developers to use speech processing measures, the E-WINDMIL in particular, to gauge the efficacy of SGCTAs. We advocate for increased industry-wide adoption of far-transfer metrics to gauge SGCTAs.

Deep learning and machine learning-based voice analysis for the detection of COVID-19: A proposal and comparison of architectures.

Alongside the currently used nasal swab testing, the COVID-19 pandemic situation would gain noticeable advantages from low-cost tests that are available at any-time, anywhere, at a large-scale, and with real time answers. A novel approach for COVID-19 assessment is adopted here, discriminating negative subjects versus positive or recovered subjects. The scope is to identify potential discriminating features, highlight mid and short-term effects of COVID on the voice and compare two custom algorithms. A pool of 310 subjects took part in the study; recordings were collected in a low-noise, controlled setting employing three different vocal tasks. Binary classifications followed, using two different custom algorithms. The first was based on the coupling of boosting and bagging, with an AdaBoost classifier using Random Forest learners. A feature selection process was employed for the training, identifying a subset of features acting as clinically relevant biomarkers. The other approach was centered on two custom CNN architectures applied to mel-Spectrograms, with a custom knowledge-based data augmentation. Performances, evaluated on an independent test set, were comparable: Adaboost and CNN differentiated COVID-19 positive from negative with accuracies of 100% and 95% respectively, and recovered from negative individuals with accuracies of 86.1% and 75% respectively. This study highlights the possibility to identify COVID-19 positive subjects, foreseeing a tool for on-site screening, while also considering recovered subjects and the effects of COVID-19 on the voice. The two proposed novel architectures allow for the identification of biomarkers and demonstrate the ongoing relevance of traditional ML versus deep learning in speech analysis.

END-TO-END NETWORK BASED ON TRANSFORMER FOR AUTOMATIC DETECTION OF COVID-19

The novel coronavirus disease (COVID-19) was declared a pandemic by the World Health Organization. The cumulative number of deaths is more than 4.8 million. Epidemiology experts concur that mass testing is essential for isolating infected individuals, contact tracing, and slowing the progression of the virus. In recent months, some machine learning methods have been proposed utilizing audio cues for COVID-19 detection. However, many works are based on hand-crafted features and deep features to detect COVID-19. There is no evidence that these features are optimal for COVID-19 detection. Therefore, we proposed an end-to-end network based on transformer for automatic detection of COVID-19. It directly learns features from the raw waveform for end-to-end learning, rather than extracting features in advance. We propose a feature extraction module to automatically extract features. And we use the transformer architectures to model the dependencies between the extracted features. It is the first end-to-end learning based on raw waveform for COVID-19 detection. Experiments on COUGHVID dataset show that our method has achieved competitive results. © 2022 IEEE

Identification of Speech Features relevant for COVID-19 Detection in Real Time Scenario

The world's biggest pandemic, COVID-19, has shown its lethal impact on human life. The current diagnostics methods are reverse transcription-polymerase chain reaction (RT-PCR) and rapid diagnostic assays have several bottlenecks in terms of the nature of sample collection as it needs some laboratory experts and careful handling of the potentially infectious samples. However, one of the non-invasive ways of diagnostics is to focus on speech modality, which has been paid less attention, during the detection of COVID-19. Hence in this work, the speech features, particularly temporal and spectral features have been studied for COVID-19 detection. The temporal features used in this work are Short-Time Energy, Long-Term Log Energy Variation (LTLEV) Zero Crossing Count (ZCC) and Pitch etc. On the other hand, the spectral features used herein are Power Spectral Density, Average Power, Mel-Frequency Cepstral Coefficients, Group delay spectrum, Spectral Entropy etc. Such spectral and temporal speech features have not been analyzed in the identification of COVID-19 symptoms to the best of authors knowledge. Further, this paper has shown the impact of COVID-19 on a real time human voice, analyzed using speech processing techniques, and shown their efficacy in detecting COVID-19. These features are safe, comparatively faster, cost-effective, and require fewer complexities. Our article will motivate the scientific community to use such features for further research in the collective battle against COVID-19. © 2022 IEEE.